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Abstract 

Random forest (RF) has been shown to outperform many other classifiers on a variety of datasets, yet 
its restriction of recursive partitions of the feature space to be axis-aligned is suboptimal in many cases. 
Several studies have proposed “oblique” decision forest methods to address this limitation. However, these 
methods either aren’t well-adapted to problems in which the number of irrelevant features are overwhelm¬ 
ing, have a time and space complexity significantly greater than RF, or require additional hyperparameters 
to be tuned, rendering training of the classifier more difficult. In this work, we establish a generalized forest 
building scheme, random projection forests. Random forests and many other currently existing decision 
forest algorithms can be viewed as special cases of this scheme. With this scheme in mind, we propose 
a special case which we call randomer forests (RerFs). We demonstrate that RerF emprically outperforms 
RF and another recently proposed oblique forest method in simulations and on benchmark data, and also 
show that it scales comparably to RF in terms of time and space complexity. Lastly, we propose two 
augmentations to RerF with the goal of making it more robust to affine transformations and outliers. We 
conclude that RerF is a competitive alternative to RF and other oblique forests. Open source code will be 
available. 

I Introduction 

Data science is becoming increasingiy important as our abiiity to coiiect and process data continues to 
increase. Supervised iearning—the act of using predictors to make a prediction about some target data—is 
of speciai interest to many appiications, ranging from science to government to industry. Ciassification, a 
speciai case of supervised iearning, in which the target data is categoricai, is one of the most fundamentai 
iearning probiems, and has been the subject of much study. A simpie pubmed search for the term “ciassifier” 
reveais neariy 10,000 pubiications, and a simiiar arXiv search reports approximateiy 1000 hits. One of the 
most popuiar and best performing ciassifiers is random forests (RFs) [3]. Two recent benchmark papers 
assess the performance of many different ciassification aigorithms on many different datasets [4, 6], and 
both conciuded the same thing: RFs are the best ciassifier. 

RF typicaiiy refers to Breiman’s Forest-IC, which uses axis-paraiiei [7], or orthogonai trees [10]. That is, the 
feature space is recursiveiy spiit aiong directions paraiiei to the axes of the feature space. Thus, in cases 
in which the ciasses seem inseparabie aiong any singie dimension, RF may be suboptimai. To address 
this, Breiman aiso proposed and characterized Forest-RC, which used iinear combinations of coordinates 
rather than individuai coordinates, to spiit aiong. Others have studied severai different variants of “obiique” 
decision forests, inciuding efforts to iearn good projections [7, 12], using principai components anaiysis 
to find the directions of maximai variance [P, 11], or directiy iearning good discriminant directions [10]. 
Another recentiy proposed method, caiied random rotation RF, uniformiy randomiy rotates the data for every 
decision tree in the ensembie prior to inducing the tree [2]. Whiie aii of these recent approaches deai with 
rotationai invariance, they faii to address two important issues. First, in reai worid data, the proportion of 
features that are irreievant is often iarge, especiaiiy for high dimensionai data, giving rise to optimai decision 
boundaries that are sparse. In such cases, we say that the signai is compressive. To our knowiedge, aii 
of the proposed obiique decision forests, with the exception of Forest-RC if tuned appropriateiy, do not 
impose a constraint on the sparsity of recursive partitions of the feature space. Therefore, such methods 
may be suboptimai in cases in which the signai is compressive. Second, as our abiiity to coiiect and 
process data continues to increase, we are encountering increasingiy massive datasets. Therefore, time- 
and space-efficient methods are becoming more and more important. With the exception of Forest-RC, 
aii obiique methods require expensive computation and/or storage. There is therefore a gap that caiis for 
the construction of a scaiabie obiique decision forest ciassifier that can perform weii in the presence of an 
overwheiming number of irreievant features. It is true that Forest-RC fits this description. However, the 
number of features that are iineariy combined to generate candidate spiits is fixed across aii nodes of every 
tree. Such a construction requires an additionai hyperparameter to be tuned, and aiso iimits the diversity of 
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trees. 


II RANDOM PROJECTION FORESTS 


To address this, we first state a generaiized forest buiiding scheme, random projection forests, which in- 
ciudes aii of the above aigorithms as speciai cases. This enabies us to formaiiy state our objective, and 
provides a iens that enabies us to propose a few novei speciai cases, which we refer to as “Randomer 
Forests" (or RerFs for short), for reasons that wiii become ciear in the sequei. We both theoreticaiiy and 
empiricaiiy demonstrate that our methods have the same time and space compiexity as random forests, 
and show on simuiated datasets that it performs weii in the face of many irreievant features and when axis- 
aiigned spiits are suboptimai. Additionaiiy, we demonstrate that our method empiricaiiy outperforms both 
RF and random rotation RF (RotRF), which is another obiique method [2], on a suite of benchmark datasets. 
Lastiy, we propose two augmentations to RerF for making the method more robust to affine transformations 
and outiiers. To conciude, we propose our method as a competitive aiternative to RF and other obiique 
decision forest methods. Open source code wiii be made avaiiabie. 

II Random Projection Forests 

Let = {{Xi, Ui) : i e [n]} be a given dataset, where e K*’ and yi € y = {yi, ■ ■ ■, yc}- A ciassification 
forest, gi^X-V^) is an ensembie of L decision trees, each tree g\X\V^) is trained on a (sub)set of the data, 
c P"). Random Projection forests are a speciai case of ciassification forests that subsume aii of the 
strategies mentioned above (see Pseudocode 1). The key idea of aii of them is that at each node of the 
tree, we have a set of predictor data points, X = e , where ST = | 5 L| is the cardinaiity 

of the set of predictor data points at the {ijf^ node of the tree. We sampie a matrix A ~ /a(T’”), 
where A e possibiy in a data dependent fashion, which we use to project the predictor matrix X 
onto a iower dimensionai subspace, yieiding X = A^X e where d < p '\s the dimensionaiity of the 

subspace. To wit, Breiman’s originai Forest-iC aigorithm can be characterized as a speciai case of random 
projection forests, in particuiar, in Forest-iC one constructs A such that for each of the d coiumns, we 
sampie a coordinate (without repiacement), and put a 1 in that coordinate, and zeros eisewhere. Breiman’s 
Forest-RC constructs A by sampiing a fixed number of coordinates (without repiacement) for each of the 
d coiumns, and puts a vaiue uniformiy sampied from [-1,1] in each of those coordinates. Simiiariy, Fio’s 
rotation forests construct A from the top d principai components of the data X at a given node. Thus, the 
key difference in aii these approaches is the choice of /a. 


Pseudocode 1 Psuedocode for Random Projection Forests, which generaiizes a wide range of previousiy 
proposed decision forests. 

Input: data: = {Xi,yi) e (Mp x y) for i e [n], tree ruies (stopping criteria, ruies for sampiing data 

points per tree, etc.), distributions on dxp matrices: A ~ fA{T)'^), preprocessing ruies 
Output: decision trees, predictions, out of bag errors, etc. 

Preprocess according to ruie 
for each tree do 

Subsampie data to obtain {X, y), the set of data points to be used in this tree 
for each ieaf node in tree do 

Let X = A'^X e R’^^'T where A ~ fA^'^) 

Find the “best” spiit coordinate k* in X and “best” spiit vaiue t*{k*) for this coordinate 
Spiit X according to whether X{k) > t*{k*) 

Assign each chiid node as a ieaf or terminai node according to stopping criteria 

end for 
end for 

Prune trees according to ruie 


The goai of this work is to find a random projection forest that shares RF’s scaiabiiity and abiiity to perform 
weii when many irreivant predictors are present, yet is abie to find decision boundaries that are iess biased 
by geometricai constraints (i.e. not constrained to be axis-aiigned), by changing the distribution Ja- The 
resuit we caii randomer forests (or RerFs for short). 
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IV EXPERIMENTAL RESULTS 


Our choice in Ja is based on the foiiowing three ideas: 

1. Whiie RF empiricaiiy performs weii in many settings, it is quite restrictive in that candidate spiits 
evaiuated at each node are constrained to be axis-aiigned. it is sometimes the case that axis-aiigned 
spiits are suboptimai, and obiique spiits may be desired. 

2. it is often the case that the number of relevant features is substantiaiiy smaiier than the number of totai 
features. We say that the signai is compressive in such cases. RF often does weii when the signai 
is compressive, which may be attributed, in part, to the extreme sparsity constraint on the coiumns 
of A. Aii of the obiique decision forest aigorithms to date, with the exception of Forest-RC, do not 
impose a sparsity constraint on the candidate spiit directions, which may cause such aigorithms to 
underperform when the signai is compressive. 

3. Except for Breiman’s Forest-RC, existing obiique decision forest aigorithms invoive expensive com¬ 
putations to identify and seiect spiits, rendering them iess space and time efficient than Forest-iC or 
Forest-RC. An obiique decision forest having a space and time compiexity comparabie to Forest-iC or 
Forest-RC is desirabie. 

To this end, we empioy very sparse random projections [9]. Rather than sampiing d non-zero eiements 
of A and enforcing that each coiumn gets a singie non-zero number (without repiacement) as RF does, we 
reiax these constraints and seiect d non-zero numbers from {-1, -i-l} with equai probabiiities and distribute 
uniformiy at random in A. 

it is apparent that RerF bares a resembiance to Forest-RC, yet there is one key difference, in Forest-RC, 
the number of nonzeros in each coiumn of A is fixed to be the same across A and for every spiit node, and 
its optimai vaiue has to be found through parameter seiection. Our construction circumvents seiection of 
this parameter by randomizing the number of nonzeros in each coiumn of A. Furthermore, our aigorithm 
aiiows A to have coiumns of varying sparsity, which may promote more diversity in the ensembie. To our 
knowiedge, this property does not exist in any of the proposed random projection forest aigorithms. Lastiy, 
we note that our construction of A preserves distances between points with high probabiiity [9]. 

IV Experimental Results 

IV.A Simulations Involving Compressive Signals 

Many ciassification probiems arise in which the signai is compressive and the optimai spiit directions are not 
axis-aiigned. We constructed two synthetic datasets with both of these properties to compare ciassification 
performance and training time of RF, RerF, and RotRF: 

Sparse parity is a variation of the noisy parity probiem. The noisy parity probiem is a muitivariate gener- 
aiization of the noisy XOR probiem and is one of the hardest constructed binary ciassification probiems. 
in the noisy parity probiem, a given sampie has a mean whose eiements are Bernouiii sampies with prob¬ 
abiiity 1/2, and then Gaussian noise is independentiy added to each dimension with the same variance. 
A sampie’s ciass iabei is equai to the parity of its mean. Sparse parity is an adaption of this probiem in 
which the sampie’s ciass iabei is equai to the parity of oniy the first p* eiements of the mean, rendering the 
remaining p- p* dimensions as noise. 

Trunk is a weii-known binary ciassification in which each ciass is distributed as a p-dimensionai muitivariate 
Gaussian with identity covariance matrices [13]. The means of the two ciasses are = (1, ..., 

and /i 2 = -pi- it foiiows from this that the signai-to-noise ratio of the ith dimension asymptoticaiiy de¬ 
creases to zero with increasing i. 

RotRF is an obiique decision forest method that has been recentiy proposed in [2]. RotRF works exactiy iike 
RF, except that the feature space is uniformiy randomiy rotated prior to inducing each decision tree. This 
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IV.A Simulations Involving Compressive Signals IV EXPERIMENTAL RESULTS 

induces oblique splits in an unstructured way, which the authors argue is favorable to structured methods 
such as in Rodriguez’s rotation forest, because such structured methods reduce the diversity of the trees. 
Uniformly random rotations of the feature space imply that in general, splits will not be sparse. Therefore, 
we conjecture that RotRF will perform increasingly poorly as the ratio of the number of irrelevant features 
to the number of relevant features becomes larger, while RF and RerF will be relatively more robust to the 
increasing presence of irrelevant features. 

Figure 1 depicts the true class posteriors in the first two dimensions of the sparse parity simulation in panel 
A and estimates of the posteriors for RF, RerF, and RotRF in panels B, C, and D, respectively. For this 
simulation, p = 20, p* = 3, and n = 1000, where p is the total number of dimensions, p* is the number of 
relevant dimensions, and n is the number of sampled data points. The number of trees used for all three 
algorithms was 1000. Various values of d, the number of columns in the random projection matrix A, were 
tried and the best for each algorithm was selected. RerF gives estimates of the posteriors closest to the 
true posteriors. 



Figure 1: Class posteriors for the sparse parity problem in the first two dimensions (refer to section IV.A for 
details). (A): True posteriors. (B) - (D): Posterior estimates for RF, RerF, and RotRF, respectively. Comparing 
panels B-D with A, it is evident that RerF produces better estimates of the posteriors than does RF or RotRF 


The left panels of Figure 2 show two-dimensional scatter plots from each of the two example simulations 
(using the first two coordinate dimensions). The middle panels show the misclassification rate relative to 
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RF against the number of observed dimensions p, for RerF and RotRF. Reiative misciassifcation rate was 
computed as the difference between the misciassification rate of either RerF or RotRF and that of RF. The 
misciassification rate of RF reiative to itseif is shown for reference. The right paneis show training time 
against the number of observed dimensions p for aii four ciassifiers. For aii comparisons, we used the same 
number of trees for each method to enabie a fair comparison. The number of trees used for each method 
in the sparse parity and Trunk simuiations were 500 and 1000, respectiveiy. In aii methods, trees were 
pruned and the minimum number of data points at a node in order to be considered for spiitting was 10. 
Gini impurity was used as the spiit criteria. The oniy parameter tuned was d, the number of candidate spiit 
directions evaiuated at each spiit node. When p < 5, each ciassifier was trained for aii d e [p]. When p> 5, 
each ciassifier was trained for aii d e Misciassification rates for each ciassifier were 

seiected as the iowest achieved from the different vaiues of d tried. Training times for each ciassifier were 
computed by averaging the training times given by aii vaiues of d tried. For sparse parity, n was fixed at 1000 
and ciassifiers were evaiuated for p e {2,5,10,25,50,100}. The reievant number of features p* was fixed at 
a vaiue of 3. For Trunk, n was fixed at 100 and RF and RerF were evaiuated tor p g {2,10,50,100,500,1000). 
RotRF was not evaiuated torp = 1000 due to computationai burden. Reiative error and training times piotted 
are the average of 25 repeated triais, and error bars represent the standard error of the mean. 

In panel B, RerF performs as well as or better than both RF and RotRF for all values of p. RotRF performs 
as well as or better than RF except for when p = 25. As conjectured, RotRF performs better than RF 
when p is small because oblique splits provide an advantage over axis-aligned splits in the sparse parity 
problem. As p increases and the ratio p* /p decreases, RerF begins to outperform RotRF. Ultimately, when 
this ratio is small enough, RotRF performs even worse than RF. RerF’s ability to perform relatively well can 
be attributed to the sparsity of oblique splits. In panel E, RerF outperforms RF for all values of p. This 
is because linear combinations of a few features can yield a higher signal-to-noise ratio than any single 
feature. RotRF outperforms both RF and RerF up to p = 100. RotRF is able to perform better than RerF in 
these cases because a larger number of features can be linearly combined to yield an even higher signal- 
to-noise ratio. When p = 500, classification performance of RotRF significantly degrades and becomes 
worse than both RF and RerF. This can be explained by the fact that when p is large enough, RotRF often 
samples linear combinations of many features each having a low signal-to-noise ratio. Such projections will 
yield a lower signal-to-noise ratio than any single feature. In panels C and F, training times are comparable 
when p is small. As panel F indicates, training time of RotRF is significantly longer than those of RF and 
RerF when p = 500. 

IV.B Theoretical Space and Time Compiexity 

For a RF, assume there are L trees. If there are n data points per tree, and each tree grows until terminal 
nodes have only 0{l) data points with p coordinates in them, there are 0{n) nodes. Then the complexity 
of constructing the random forest, disregarding cross-validation or other randomization techniques for pre¬ 
venting overfitting, is 0{Ln'^plogn). In practice the trees are shallower and stop much earlier than when 
nodes have 0(1) points, so “in practice” the complexity often appears to be 0{Lnp\ogn). Storing RF re¬ 
quires 0{Lnlogn) because each node can be represented by the index of the nonzero coordinate. The only 
additional space constraint is storing which indices are positive, and which are negative, which is merely 
another constant. 

RerF has a very similar time and space complexity, unlike many of the other oblique random forest variants. 
Specifically, assume that RerF also has L trees, and n data points per tree, and no pruning, so 0{n) 
nodes. Let Uk be the number of data points at node k of the tree. Like RF, RerF requires 0{p) time to 
sample p non-zero numbers, and 0{pnk) time to obtain the new matrix, X, because it is a sparse matrix 
multiplication, in node k with 0{nk) points. RerF also takes another O{p/nlog{p/n)) = 0{T) time to find the 
best dimension. Thus, in total, in the case of well-balanced trees, RerF also requires only 0{LprE logn) time 
to train. To store the resulting RerF, like RF, requires 0{Lnlogn), because each node can be represented by 
the indices of the coordinates that received a non-zero element, and the expected number of such indices 
is 0(1). Note that these numbers are in stark contrast to other oblique methods. RotRFs, in particular, 
require a QR decomposition having a time complexity of 0{p^) in order to generate random rotation matrix 
for each tree. Rotating the data matrix prior to inducing each tree additionally requires 0{np^). Therefore, 
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Figure 2: Sparse parity (A-C) and Trunk (D-F) simuiations (see section IV.A for detaiis). (A) and (D): 
Scatterpiots of sampied points in the first two dimensions. (B) and (E): Error rates of RerF and RotRF 
reiative to RF across different vaiues of p. (C) and (F): Same as (B) and (E) except absoiute training time 
is piotted on the y-axis instead. The cyan, green, and magenta iines correspond to RF, RerF, and RotRF, 
respectiveiy. This coior coding is adopted in figures 3 and 4 that foiiow. Both RerF and RotRF do better 
than RF when the number of irreievant features is sufficientiy smaii, due to their abiiity to generate obiique 
parititions. However, when the number of irreievant features becomes iarge enough, performance of RotRF 
rapidiy degrades. Training times show that RerF scaies identicaiiy with RF, whiie RotRF scaies pooriy with 
iarge p (note that in paneis E and F, RotRF is oniy piotted up to p = 500 due to computationai constraints. 


RotRF becomes very expensive to compute when p is iarge. This can expiain the trend seen in Figure 
2(F). In addition to storing aii of the decision trees, RotRF has to store a rotation matrix for each tree, which 
requires 0{p'^). 

IV.C Effects of Transformations and Outliers 

We next want to compare the robustness of RF, RerF, and RotRF to various data transformations. To do 
so, we consider severai different modifications to the simuiation settings described in the previous section: 
rotation, scaie, affine, and outiiers. To rotate the data, we simpiy generate rotation matrices uniformiy 
and appiy them to the data. To scaie, we appiied a scaiing factor sampied from a uniform distribution on 
the intervai [0,10] to each dimension. Affine transformations were performed by appiying a combination 
of rotations and scaiings as just described. Additionaiiy, we examined the effects of introducing outiiers. 
Outiiers were introduced by sampiing points from the distributions as previousiy described but instead using 
covariance matrices scaied up by a factor of four. Empiricaiiy, an addition of 20 points from these outiier 
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IV.D Benchmark Data IV EXPERIMENTAL RESULTS 

models to the original 100 points was found to produce a noticeable but not overwhelming effect on classifier 
performance. 

Figure 3 shows the effect of these transformations and outliers on the sparse parity (panels A-E) and Trunk 
(panels F-J) problems. Panels A-E show that RerF and RotRF outperform RF in both untransformed and 
transformed representations for Sparse Parity. This makes sense because linear combinations of variables 
can have some information about the class, whereas the original features are all uninformative on their 
own. For the trunk simulations, panel G shows that RotRF is more robust to rotations (as it should be), but 
is fragile in the face of affine transformations. 



Figure 3: The effects of different transformations applied to the sparse parity (A-E) and Trunk (F-J) sim¬ 
ulations on classification performance (see section IV.C for details). Specifically, we consider rotations, 
scalings, affine transformations, as well as the addition of outliers. RerF and RotRF outperform RF on 
sparse parity. The trunk simulations demonstrate clearly that RotRF, while invariant to rotation, is sensitive 
to affine transformations. 


IV.D Benchmark Data 

In addition to the simulations, RF, RerF, and RotRF were evaluated on 115 of the 121 datasets as described 
in p]. The six remaining datasets were not used because their high dimensionality and large number of 
data points rendered the classifiers both time and space costly, particularly for RotRF. As in the previous 
section, transformations, with the exception of outliers, were applied to the datasets to observe their affects 
on performance of the three algorithms. Classifiers were trained on the entire training sets provided. For 
each data set, misclassification rates were again estimated by out of bag error. The number of trees used 
in each algorithm was 1000 for datasets having at most 1000 data points and 500 for datasets having 
greater than 1000 data points. All algorithms were trained using five different values for d, and error rates 
for each algorithm were selected as the minimum given by the five. For each dataset, relative performance 
ratios for each algorithm were computed by dividing the error rate of each algorithm by the minimum error 
rate of the three. The empirical cumulative distribution functions of relative performance ratios for each 
algorithm were computed and plotted in panels A-D of Figure 4. Such plots are called performance profiles 
[5]. Performance profiles are useful in visualizing how frequently a particular algorithm wins, and when it 
loses, how frequently it loses by a certain amount. Areas under the curves (AUCs) were computed for each 
performance profile. This metric is valuable in that it doesn’t necessarily indicate how frequently a particular 
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IV.E Rank Transforming the Data and Fast Supervised Projections IV EXPERIMENTAL RESULTS 
algorithm is the best. Rather, it is an indicator of robustness. For instance, it is possible that an algorithm 
that wins on the majority of datasets can have a lower AUC than the other algorithms if, when it loses, 
it frequently loses by a large margin. On the original benchmarks (Figure 4 panel A), RerF achieves the 
largest AUC, followed by RF and lastly RotRF. The same trend is seen when scaling (panel C) and affine 
transformations (panel D) are applied. When the datasets are rotated, RotRF achieves the largest AUC, 
followed by RerF, and lastly RF. 

Since the simulations in section IV.A revealed that the speed of RF and RerF scaled comparably, we wanted 
to see if this observation held on the benchmark data. In the same fasion as the left panels of Figure 4, we 
plotted performance profiles for training times in the right panels. Across all transformations, we see that 
RF and RerF perform comparably in terms of speed. Panels A and C show that RotRF performs the worst 
on the original benchmark data and the randomly scaled benchmark data. It performs well, however, on 
the rotated and affine transformed data. One possible explanation for this is that when the data is rotated, 
RF and RerF have a more difficult time finding good split directions, which could result in deeper trees (and 
subsequently more time spent in training). 


Untransformed 


Rotated 


Scaled 


Affine 




Figure 4: Classification (panels A-D) and speed (panels E-H) performance profiles on benchmarks with 
various transformations applied (see section IV.D for details). AUC denotes the area under the curve. In 
terms of classification accuracy, RerF outperforms RF in all five data settings and only loses to RotRF when 
the data is rotated. RF and RerF have comparable training time performance, while RotRF tends to be 
slower. 


IV.E Rank Transforming the Data and Fast Supervised Projections 

When dimensions of X are linearly combined, we lose scale and unit invariance. We therefore note that 
random forests have a special property: they are invariant to monotonic transformations of the data applied 
to each coordinate in the ambient (observed) space. They are effectively operating on the order statistics, 
rather than actual magnitudes of coordinates. In other words, if we convert for each dimension the values 
of the samples to their corresponding ranks, random forests yield the exact same result. Therefore, for our 
second trick, we adopt the same policy, and “pass to ranks”, or rank transform, prior to doing anything 
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V CONCLUSION AND FUTURE WORK 


else. We call RerFs that pass to ranks RerF(r). 

Additionally, there Is evidence that using the data to construct A at each node can significantly Improve 
performance [7], However, previously proposed approaches for using the data require methods that are 
time- and space-intensive compared to standard random forests. We propose a simple strategy: “compute 
the mean difference vectors”. In other words, given a two-class problem, let 6 = fio- Fi, where /tc Is the 
estimated class conditional mean. Under a spherically symmetric class conditional distribution assumption, 
S Is the optimal projection vector. When there are C > 2 classes, the set of all pairwise distances between 
fie and fic' Is of rank C- 1. Because RerFs are approximately rotatlonally Invariant, we can simply compute 
all the class conditional means, and subtract each from one of them (vy^e chose the fic for which nc, the 
number of samples In that class. Is the largest). Thus, we construct X by concatenating A'^ with A = 
(i5(2) - ^( 1 ),... ,(5(c) - (5(1)), where Is the 6 vector for the class with the largest number of samples, to 
obtain A^. Then we compute X = A^'X. Computing this matrix Is extremely fast, because It does not rely 
on costly singular value decompositions, matrix Inversions, or convex problems. Thus, It nicely balances 
using the data to find good vectors, but not using much computational space or time. Furthermore, It also 
provides a set of dense projections to supplement the sparse projections of RerF. Denote RerFs that Include 
this matrix RerF(d), and If they pass to ranks first, RerF(d-i-r). 

Figure 5 demonstrates the utility of passing to ranks and using supervised projections on various transfor¬ 
mations of the Trunk simulation. Panels A and B again show that RF and RerF are sensitive to rotation 
and therefore affine transformations. Panel C shows that RerF(d) achieves a lower error rate than RerF In 
the untransformed setting and Is not sensitive to rotation. However, It Is sensitive to scale and therefore 
affine transformations. Panel D shows that RerF(d-i-r) Is more robust to affine transformations than the 
other methods, as hoped. Panel E again showsthat RotRF Is sensitive to scale and subsequently affine 
transformations. 


RF RerF RerF(d) RerF(d+r) Rotation RF 



Untransformed 
— — Rotated 
Scaled 
Affine 


Figure 5: The utility of supplementing RerF with supervised projections and rank transforming the data 
prior to Inducing trees (see section IV.E for details). The effect of various transformations on classification 
performance of RF, RerF, RerF(d), RerF(dH-r), and RotRF for the Trunk simulation are shown In panels A- 
E, respectively. Panel D shows that RerF(d-i-r), which supplements RerF with supervised projections, and 
passes the data to ranks. Is more robust to affine transformations than the other methods. 


V Conclusion and Future Work 

We have proposed novel methods for constructing decision forests, which we call RerFs. We view these 
methods as special cases of a more general random projection forest, which Include Brelman’s original 
Forest-IC and Forest-RC, as well as previously proposed oblique decision forests. We have demonstrated 
In simulations that RerFs are especially well-suited for classification problems In which axis-parallel splits 
are suboptimal, and at the same time, have a large number of Irrelevant features relative to relevant ones. 
This could explain RerF’s excellent empirical performance on a suite of over 100 benchmark datasets, as 
real data often has the properties just described. Moreover, RerF, unlike other oblique decision forests, 
preserves the time and space complexity of Forest-IC, and Is only marginally sensitive to scaling. Lastly, 
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we propose two additional modifications. One involves supplementing sparse random projections with a 
particular form of fast supervised projections in the construction of candidate split directions. The other 
involves passing the data to ranks with the goal of making the classifier more robust to outliers. 

Much work is still to be done with our proposed methods. The simplicity of RerFs suggest that they will 
be amenable to theoretical investigation, extending the work of Biau et al. (2008) to the RerF setting [ ]. 
Moreover, we hope that theoretical investigations will yield more insight into which distributions will 

be optimal under different distributional settings, both asymptotically and under finite sample assumptions. 
Even under the currently proposed /a(T’„), our implementation has room for improvement. Although it has 
the same space and time complexity as RF, we will determine explicit constants, and improve our imple¬ 
mentation accordingly. Indeed, our current implementation is a proof-of-concept MATLAB implementation. 
We will utilize recent GPU and semi-external memory methods to significantly speed up RerF [14]. As with 
all decision forests, multiclass classification is but one exploitation task they can be used for; therefore, we 
will also extend this work to enable regression, density estimation, clustering, and hashing. We will provide 
open source code to enable more rigorous and extended analyses by the machine learning community. 
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